Image processing apparatus, image processing method, storage medium, system, and electronic apparatus

ABSTRACT

An image processing apparatus includes processing circuitry. The processing circuitry is configured to detect a positional shift amount of each of a plurality of images; select a composite target image from the plurality of images based on the detected positional shift amount; and obtain a composite image based on the positional shift amount and the selected composite target image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation Application of U.S. application Ser.No. 16/217,536, filed Dec. 12, 2018, which is based on and claimspriority pursuant to 35 U.S.C. § 119(a) to Japanese Patent ApplicationNo. 2017-239525, filed on Dec. 14, 2017 and Japanese Patent ApplicationNo. 2018-230626, filed on Dec. 10, 2018, in the Japan Patent Office, theentire disclosures of which are hereby incorporated by reference herein.

BACKGROUND Technical Field

Embodiments of the present disclosure relate to an image processingapparatus, an image processing method, an electronic apparatus, and anon-transitory recording medium storing program code for causing theimage processing apparatus to perform a method for processing an image.

Background Art

An image-capturing technique called multi-shot compositing is known thatobtains a composite image by compositing a plurality of images capturedwhile moving (minutely vibrating) the image sensor on a pixel-by-pixelbasis. With such a multi-shot compositing, higher-definition images(higher image quality, higher accuracy) than typical one-shot images canbe obtained.

SUMMARY

In one aspect of this disclosure, there is provided an improved imageprocessing apparatus including processing circuitry. The processingcircuitry is configured to detect a positional shift amount of each of aplurality of images, select a composite target image from the pluralityof images based on the detected positional shift amount, and obtain acomposite image based on the positional shift amount and the selectedcomposite target image.

In another aspect of this disclosure, there is provided an improvedmethod of processing an image. The method includes detecting apositional shift amount of each of a plurality of images; selecting acomposite target image from the plurality of images based on thedetected positional shift amount; and obtaining a composite image basedon the positional shift amount and the selected composite target image.

In still another aspect of this disclosure, there is provided animproved non-transitory recording medium storing a program for causing acomputer to execute the above-described method.

In yet another aspect of this disclosure, there is provided an improvedsystem including processing circuitry. The processing circuitry isconfigured to detect a positional shift amount of each of a plurality ofimages, select a composite target image from the plurality of imagesbased on the detected positional shift amount, and obtain a compositeimage based on the positional shift amount and the selected compositetarget image.

In further another aspect of this disclosure, there is provided animproved electronic apparatus including the above-described imageprocessing apparatus and an image-capturing device configured to capturethe plurality of images.

BRIEF DESCRIPTION OF THE DRAWINGS

The aforementioned and other aspects, features, and advantages of thepresent disclosure will be better understood by reference to thefollowing detailed description when considered in connection with theaccompanying drawings, wherein:

FIG. 1 is a block diagram of a schematic configuration of an electronicapparatus equipped with a camera unit according to a first embodiment ofthe present disclosure;

FIGS. 2A, 2B, 2C, and 2D are illustrations for explaining examples of amulti-shot composite mode;

FIG. 3 is a functional block diagram of the image processing apparatus(processor) according to a first embodiment of the present disclosure;

FIG. 4A is an illustration of an example reference image that is one offour images continuously captured;

FIGS. 4B, 4C, and 4D are illustrations of remaining three images of thefour images set as comparative images;

FIG. 5 is a conceptual diagram for explaining a composite image obtainedby compositing a reference image and comparative images;

FIG. 6 is a flowchart of image processing performed by the imageprocessing apparatus according to an embodiment of the presentdisclosure;

FIG. 7 is a flowchart of a process of selecting a composite target imageperformed by a selecting unit, according to an embodiment of the presentdisclosure;

FIG. 8 is a flowchart of a process of selecting a composite target imageperformed by a selecting unit according to another embodiment of thepresent disclosure;

FIG. 9 is a functional block diagram of an image processing apparatus(processor) according to a second embodiment of the present disclosure,illustrating the internal structure of the image processing apparatus;

FIGS. 10A, 10B, 10C, and 10D are illustrations of an example in which aplurality of images is divided into a predetermined number of imageareas;

FIG. 11 is a flowchart of an example of an image-capturing processingaccording to the second embodiment of the present disclosure;

FIGS. 12A and 12B are a rear view and a cross-sectional view,respectively, of an example configuration of a vibration-proof unit;

FIG. 13 is a rear view of a movable stage of the vibration-proof unit;

FIG. 14 is an enlarged cross-sectional view of an X drive unit includingan X-direction magnet and an X-drive coil;

FIG. 15 is an enlarged cross-sectional view of a Z drive unit includinga Z-direction magnet, a Z-drive coil, and a Z-direction Hall element;

FIGS. 16A and 16B are illustrations for explaining adverse effects ofimage blur in the rotational direction within an XY plane; and

FIG. 17 is an illustration of an example in which a plurality of imagesis divided in to image areas having different sizes.

The accompanying drawings are intended to depict embodiments of thepresent disclosure and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted.

DETAILED DESCRIPTION

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this patent specification is not intended to be limited to thespecific terminology so selected and it is to be understood that eachspecific element includes all technical equivalents that have the samefunction, operate in a similar manner, and achieve similar results. Asused herein, the singular forms “a”, “an”, and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise.

Although the embodiments are described with technical limitations withreference to the attached drawings, such description is not intended tolimit the scope of the disclosure and all of the components or elementsdescribed in the embodiments of this disclosure are not necessarilyindispensable.

Embodiments of the present disclosure are described in detail withreference to the drawings. Like reference numerals designate identicalor corresponding components throughout the several views and adescription of same is omitted.

First Embodiment

FIG. 1 is a block diagram of a hardware configuration of an electronicapparatus 1 equipped with a camera unit. The electronic apparatus 1includes an image processing apparatus according to a first embodimentof the present disclosure, and is capable of performing an imageprocessing method according to program code stored in a non-transitoryrecording medium.

Examples of the electronic apparatus 1 include various kinds ofapparatuses equipped with an image-capturing capability such as adigital camera, a mobile phone, and a game machine. In the embodimentsof the present disclosure, examples in which the electronic apparatus 1is a digital camera are described. Alternatively, the electronicapparatus 1 may be various other apparatuses such as personal computers(PCs) that receive an image and performs image processing on the image,without an image-capturing capability. Further, the electronic apparatus1 according to the embodiments of the present disclosure is capable ofexecuting multi-shot compositing using camera shake. Accordingly, theelectronic apparatus 1 is preferably a portable electronic apparatusthat easily causes camera shake, having the image-capturing capability.

The digital camera as the electronic apparatus 1 includes, inside acamera body CB, a camera unit (image-capturing device) 10, an imageprocessing apparatus (processor) 20, a memory (for example, a randomaccess memory (RAM)) 30, a recording medium (for example, USB memory)40, a display device (display) (for example, a liquid crystal display(LCD)) 50, an input device (switch) 60, a sensor 70, a vibration-proofunit (camera shake correction device) 80, and a central processing unit(CPU) 90, which are directly or indirectly connected to each other via abus 100. Note that the image processing apparatus (processor) 20 and theCPU 90 may be configured by the same hardware device or may beconfigured as separate hardware device.

The camera unit 10 has an imaging optical system and an image sensor 11(FIG. 2). The imaging optical system forms an image of an object (objectimage) on the light-receiving surface of the image sensor 11, and theimage sensor 11 converts the formed image into electrical signals usinga plurality of pixels having different detection colors arranged in amatrix. The electrical signals are then transmitted to the imageprocessing apparatus 20 as an image. The image processing apparatus 20performs predetermined image processing on the image captured by thecamera unit 10. The image processed by the image processing apparatus 20is temporarily recorded in the memory 30. The image recorded in thememory 30 is stored in the recording medium 40 in accordance with theselection and determination by the user and displayed on the displaydevice 50.

The input device 60 includes, for example, a power switch, a releaseswitch, a dial for selecting and setting various functions, a four-wayswitch, and a touch panel. The sensor 70 includes, for example, anacceleration sensor, an angular velocity sensor, and an angularacceleration sensor for detecting the acceleration, the angularvelocity, and the angular acceleration of the body of the digital camera(the electronic apparatus 1), respectively. The output of the sensor 70is transmitted to the CPU 90 as a shake detection signal indicatingshaking of the body of the digital camera.

The vibration-proof unit 80 moves at least one of the imaging opticalsystem and the image sensor 11 of the camera unit 10, as a moving member(drive member), in a direction different from the direction of theoptical axis of the imaging optical system (for example, within a planeorthogonal to the optical axis of the imaging optical system). The CPU90 controls driving of the vibration-proof unit 80. The CPU 90 receivesa shake detection signal indicating shaking of the body of the digitalcamera from the sensor 70 and causes the vibration-proof unit 80 to movethe moving member in a direction different from the direction of theoptical axis of the imaging optical system. With such a configuration,the image-forming position of the object image is shifted on the imagesensor 11 so that the image blurring due to camera shake can becorrected. The configuration of the vibration-proof unit 80 will bedescribed later in detail.

The digital camera (the electronic apparatus 1) operates in a shootingmode (multi-shot composite mode) in which an image capturing operationis performed a plurality of times in chronological order while minutelymoving the image sensor 11 of the camera unit 10 in a directiondifferent from the direction of the optical axis of the imaging opticalsystem (for example, within a plane orthogonal to the optical axis ofthe imaging optical system) using the vibration-proof unit 80. In theshooting mode (image-capturing mode, multi-shot composite mode), thedigital camera further composites these images to obtain one compositeimage (which is obtained not by simply adding the images but byprocessing image data using special calculations), thus generating asuper high-definition (high-quality) image. Unlike the typical compositetechnology that obtains one-color information for each pixel, in themulti-shot composite mode according to the embodiments of the presentdisclosure, color information regarding red, green, and blue (RGB) foreach pixel is obtained to draw a high-definition image with more detailand better color reproduction. Further, in the multi-shot composite modeaccording to the embodiments of the present disclosure,higher-sensitivity noise can be reduced without generating moire andfalse color.

FIGS. 2A, 2B, 2C, and 2D are illustrations for explaining an example ofa multi-shot composite mode. In FIGS. 2A to 2D, the image sensor 11includes a large number of pixels arranged at a predetermined pixel inmatrix on a light-receiving surface. One of the Bayer-array colorfilters R, G (Gr and Gb), and B is disposed on the front surface of eachpixel. Each pixel detects the color of an object light beam that haspassed through the color filter R, G (Gr, Gb), or B on the front surfaceand hit the same pixel. That is, each pixel photoelectrically convertslight of a color component (color band) into an electrical signal andaccumulates electrical charge according to the intensity (luminance) ofthe light. More specifically, one image is captured at the referenceposition of FIG. 2A, and another image is captured at a position towhich the light flux region surrounded by the thick frame has been moveddownward by one pixel relative to the image sensor 11 as illustrated inFIG. 2B. Further, still another image is captured at a position (FIG.2C) to which the light flux region surrounded by the thick frame hasbeen further moved by one pixel from the position of FIG. 2B to theright relative to the image sensor 11. Then, yet another image iscaptured at a position (FIG. 2D) to which the light flux regionsurrounded by the thick frame has been further moved upward from theposition of FIG. 2C by one pixel, relative to the image sensor 11.Finally, the light flux region returns to the reference position in FIG.2A. In such a manner, four images are captured in chronological orderwhile moving (driving) the light flux region surrounded by the thickframe one pixel at a time relative to the image sensor 11 to draw asquare within the plane orthogonal to the optical axis. Then, thecaptured four images are transmitted as raw image data to the imageprocessing apparatus 20. The image processing apparatus 20 compositesthe four images captured in chronological order by the image sensor 11to obtain a composite image.

In the multi-shot compositing using the vibration-proof unit 80, thebody of the digital camera is attached to, for example, a tripod, so asto reliably move the light flux region on a pixel-by-pixel basis on theimage sensor 11. In the electronic apparatus 1 according to theembodiments of the present disclosure, the multi-shot compositing isexecutable without using the vibration-proof unit 80 (without camerashake correction) and also executable with the body of the digitalcamera (the electronic apparatus 1) held by the user (photographer). Inother words, the electronic apparatus 1 according to the embodiments ofthe present disclosure obtains a composite image by the multi-shotcompositing based on an image misalignment (shift) for each shot due tocamera shake (fluctuation) of the photographer, instead of activelymoving the image sensor 11. Hereinafter, this shooting (capturing) modeis sometimes called “camera shake multi-shot composite mode”.

By operating the input device (switch) 60 of the digital camera, theshooting mode is switched between the camera shake multi-shot compositemode (a given shooting mode) and another shooting mode (for example, themulti-shot composite mode using the vibration-proof unit 80). Further,the display device (display) 50 of the digital camera is capable ofdisplaying the setting status indicating that the camera shakemulti-shot composite mode (given shooting mode) is set. In addition tothe camera shake multi-shot composite mode, the given shooting mode(particular image processing mode) according to the embodiments of thepresent disclosure includes a shooting mode (an image processing mode)in which a plurality of images with similar composition, angle,photographing time, and image quality are selected/extracted from aplurality of images continuously shot without camera shake, or from adesignated folder and cloud storage in which a set of recorded imagessuch as moving images are stored. The input device (switch) 60 and thedisplay device (display) 50 of the digital camera enable the givenshooting mode (particular image processing mode) to be set anddisplayed.

FIG. 3 is a block diagram of a functional configuration of the imageprocessing apparatus (processor) 20 including a matching unit 21, adetecting unit 22, a selecting unit 23, and a compositing unit 24. Thesefunctional units are implemented by the CPU 90 according to an imageprocessing program.

The matching unit 21 determines whether a plurality of images capturedby the camera unit 10 are suitable for multi-shot compositing using, forexample, a method of evaluating the degree of match between images (forexample, pattern matching) or based on the output of the sensor 70.There are degrees of freedom for the number of images, and no specificnumber is set. For example, the matching unit 21 is capable of executinga matching process when a predetermined number of images (for example,four images) are received by the matching unit 21. For example, when theplurality of images is continuously captured with the same compositionand the same angle using the continuous shooting mode (for example, theobjects of these images are the same and correlate), the matching unit21 is more likely to determine that this particular plurality of imagesis suitable for the multi-shot compositing. However, when a plurality ofimages is captured with different composition and angles in a staggeredmanner (for example, the objects of the images are different from eachother and are not correlated), the matching unit 21 is more likely todetermine that this particular plurality of images is not suitable forthe multi-shot compositing. When the matching unit 21 determines thatthe plurality of images is suitable for the multi-shot compositing, theimage processing apparatus 20 continues executing the multi-shotcompositing. When the matching unit 21 determines that the plurality ofimages is not suitable for the multi-shot compositing, the imageprocessing apparatus 20 ends the multi-shot compositing process.

Preferably, the plurality of images, which may include a referenceimage, comparative images, and a composite target image to be describedlater, exhibits mutual correlation between pixels. For example, theplurality of images is a moving image or continuously shot images inwhich there is no significant change in an object to be captured and thedegree of exposure. When the object is a still object such as alandscape or a photograph, the plurality of images is not limited to amoving image captured at one time or continuously shot images, and maybe captured in a staggered manner. Even when the object to be capturedis the same for the plurality of images, the degree of exposure differsdepending on the shooting moment such as day and night. In such a case,compositing images might result in failure, described later. To avoidsuch a situation, the plurality of images is preferably captured atsubstantially the same degree of exposure. When the plurality of imagescaptured at different degrees of exposure is used for the imagecompositing, the degrees of exposure of the images are normalized toconform to some one of the degrees of exposure of the images. Thus, theaccuracy of pattern matching increases.

When the matching unit 21 determines that the plurality of images is notsuitable for the multi-shot compositing, the camera unit 10 continuescapturing images while discarding unwanted frames until a plurality ofimages suitable for multi-shot compositing are obtained. In this case,the shooting conditions may be set (restricted) so as to make it easierto obtain such a plurality of images suitable for the multi-shotcompositing. Examples of the shooting conditions include InternationalOrganization for Standardization (ISO) sensitivity, shutter speed,aperture opening degree, focal length, shooting distance, andenvironmental brightness. Such shooting conditions may be set(restricted). Further, when the signal-to-noise ratio (S/N ratio) ispoor, the number of images shot by the camera unit 10 can be increased.

The plurality of images to be transmitted to the matching unit 21 is notlimited to images immediately after the camera unit 10 shoots. Forexample, a plurality of images with similar composition, angle, shootingtime, and image quality are selected/extracted from a designated folderor cloud storage in which a set of recorded images such as moving imagesare stored. When such a selection/extraction process is repeatedcontinuously, a plurality of images may be an independent set of Nimages or may be a set of (N−1) images including identical images.

The detecting unit 22 detects the pixel shift amount (positional shiftamount) of each of the plurality of images determined to be suitable formulti-shot compositing by the matching unit 21. The detecting unit 22may reliably and precisely detect the pixel shift amount of each of theplurality of images using the typical technique such as block matching.In addition, the detecting unit 22 is also capable of reliably andprecisely detecting the pixel shift amount of each of the plurality ofimages using various methods described below.

For example, the detecting unit 22 detects the pixel shift amount ofeach of the plurality of images based on the output of at least one ofthe acceleration detector, the angular velocity detector, and theangular acceleration detector, which constitute the sensor 70.

The detecting unit 22 detects the pixel shift amount of each of theplurality of images on a pixel to pixel basis or a sub-pixel tosub-pixel basis, based on the pixel output of the image sensor 11.Further, the detecting unit 22 detects the pixel shift amount of theplurality of images for each RGB plane based on the pixel output of theimage sensor 11. In this case, the detecting unit 22 may use only aspecific RGB plane out of the plurality of RGB planes, or may change theRGB plane to be used. For example, the detecting unit 22 uses a G(green) plane to detect the pixel shift amount between a first image anda second image, and uses a R (red) plane to detect the pixel shiftamount between a third image and a fourth image.

In some embodiments, the detecting unit 22 performs a detection processaccording to the configuration in which the detection mode using theoutput of the sensor 70 is combined with the detection mode using thepixel output of the image sensor 11 described above. In such a case, thedetecting unit 22 roughly estimates a direction in which pixels aremisaligned (shifted) using the output of the sensor 70 before accuratelydetecting the pixel shift amount using the pixel output of the imagesensor 11.

When the pixel output of the image sensor 11 includes a pixel output fora given use, the detecting unit 22 excludes the pixel output for thegiven use or assigns a low weight to the pixel output for the given usebefore detecting the pixel shift amount. The pixel output for the givenuse includes, for example, a phase difference detection pixel to be usedin operations other than the shooting operation.

The selecting unit 23 selects a composite target image from theplurality of images according to the pixel shift amount of each of theplurality of images detected by the detecting unit 22. Morespecifically, the selecting unit 23 sets any one of the plurality ofimages as a reference image and sets the remaining images as comparativeimages, and compares the reference image and each of the comparativeimages to obtain the pixel shift amount between the reference image andeach of the comparative images. Then, based on the obtained pixel shiftamounts, the selecting unit 23 selects a composite target image from thecomparative images.

FIGS. 4A, 4B, 4C, and 4D are illustrations of a reference image andthree comparative images 1 to 3 of four continuously shot images,respectively. There is a minute pixel shift (misalignment) on the orderof, for example several pixels between the reference image and each ofthe comparative images 1 to 3, which is difficult to recognize visually.This pixel shift (misalignment) is caused by, for example, camera shake(fluctuation) of the photographer holding the body of the digitalcamera. In the embodiments of the present disclosure, the pixel shiftbetween the reference image and each of the comparative images 1 to 3 isregarded as the amount of drive of the image sensor 11 (see FIGS. 2A to2D) in the multi-shot compositing using the vibration-proof unit 80.Accordingly, the multi-shot compositing can be achieved without usingthe vibration-proof unit 80.

When the pixel coordinates (dx, dy) of the reference image (A) isdefined as (0, 0), the selecting unit 23 searches for a combination ofthe comparative images (B, C, and D) of the pixel coordinates (dx, dy)as described below. In this case, each pixel coordinate represents thepixel shift amount of each comparative image.

(A) reference image: (dx, dy)=(0, 0)

(B) Comparative image: (dx, dy)=(even number, odd number)

(C) Comparative image: (dx, dy)=(odd number, odd number)

(D) Comparative image: (dx, dy)=(odd number, even number)

Ideally, the pixel coordinates (dx, dy) of each comparative image arerepresented by an integer number of pixels, but in fact, such cases donot exist. It is difficult to select a composite target image thatsatisfies any of the above-described (A) to (B). In view of such asituation, the selecting unit 23 sets an allowable error “Eallowable” ofone pixel or less that satisfies −0.25<Eallowable<0.25, for example, andselects a target composite image. More specifically, the selecting unit23 replaces (A) to (D) with the following (A′) to (D′) and selects atarget composite image. If the value of Eallowable is 0 in (A′) to (D′),the values of (A) to (D) are equal to the values of (A′) to (D′).

(A′) Reference image: (dx, dy)=(0, 0)

(B′) Comparative image: (dx, dy)=(even number+Eallowable, oddnumber+Eallowable)

(C′) Comparison image: (dx, dy)=(odd number+Eallowable, oddnumber+Eallowable)

(D′) Comparative image: (dx, dy)=(odd number+Eallowable, evennumber+Eallowable)

By setting the allowable error Eallowable within the range of−0.25<Eallowable<0.25, only the pixel coordinates in the range close tothe integer pixel can be picked up and so a composite target image canbe selected with high accuracy and reliability, irrespective of whetherthe number of shifted pixels is an even number or an odd number. Forexample, when a certain pixel coordinate is “−2.90” or “3.22”, theallowable error Eallowable is applied to the value. Accordingly, thepixel coordinate is determined to be close to “3”, and such a pixelcoordinate is selected. However, when a certain pixel coordinate is“−2.64” or “3.38”, it is determined to be not close to “3” even afterapplying the allowable error Eallowable to the pixel coordinate, andsuch a pixel coordinate is not selected. Note that, by setting theallowable error Eallowable within the range of −0.5<Eallowable<0.5, thepixels in the vicinity of the intermediate value between the odd pixeland the even pixel are successfully sorted into the odd pixel or theeven pixel. For example, when certain pixel coordinates are “3.48” and“3.53” in the vicinity of an intermediate value between an odd pixel andan even pixel, it is determined that “3.48” is close to “3” and “3.53”is close to“4”.

The selecting unit 23 sets the reference image that satisfies theabove-described (A′) among the plurality of images and also selects thecomparative image that satisfies at least one of the above (B′) and/orthe comparative image that satisfies the above (D′) as a compositetarget image(s). With this configuration, two types of G (Gr, Gb) can beincluded as the color-information component for each pixel to compositeimages, which is to be described later.

More preferably, the selecting unit 23 sets the reference image thatsatisfies the above (A′) among the plurality of images, and selectscomparative image that satisfies the above (B′) and the comparativeimage that satisfies the above (C′), and the comparative image thatsatisfies the above (D′) as the composite target images (in this case, aplurality of composite target images is selected from the comparativeimages). This configuration enables the RGB color-information componentsto be included in each pixel for composite images, which will bedescribed later.

Upon failing to select the composite target image from the comparativeimages, the selecting unit 23 enlarges (increases) the boundary value(absolute value) of the allowable value Eallowable and retries to selecta composite target image. For example, the selecting unit 23 sets theabsolute value of the allowable value Eallowable to 0.01 at the firstselection of the composite target image, and sets the absolute value ofthe allowable value Eallowable to 0.05 at the second selection of thecomposite target image. Further, the selecting unit 23 sets the absolutevalue of the allowable value Eallowable to 0.10 at the third selectionof the composite target image, and sets the absolute value of theallowable value Eallowable to 0.15 at the fourth selection of thecomposite target image. Still further, the selecting unit 23 sets theabsolute value of the allowable value Eallowable to 0.20 at the fifthselection of the composite target image, and sets the absolute value ofthe allowable value Eallowable to 0.25 at the sixth selection of thecomposite target image. When still failing to select the compositetarget image after setting the absolute value of the allowable valueEallowable to 0.25, the selecting unit 23 ends the process of selectingthe composite target image. By gradually increasing the boundary value(absolute value) of the allowable value Eallowable, the accuracy ofselection of the composite target image is gradually reduced.

Upon failing to select the composite target image from the comparativeimages, the selecting unit 23 resets the reference image (and thecomparative images), and retries to select the composite target image.Since the pixel shift amount is defined by the relation with thereference image, whether the composite target image is successfullyselected depends on the reference image set among the plurality ofimages. For such a reason, the composite target image may besuccessfully selected by resetting the reference image to select thecomposite target image again. That is, when a certain image is set asthe reference image and selecting the composite target image fails,another image is set as the reference image, which might enable thecomposite target image to be successfully selected.

When the pixel coordinates (dx, dy) of the comparative image, which isthe pixel shift amount, exceed a predetermined error threshold (forexample, several tens of pixels), the selecting unit 23 ends the processof selecting the composite target image.

The compositing unit 24 obtains a composite image based on the pixelshift amount detected by the detecting unit 22 and the composite targetimage (the composite target image selected based on the reference imageand the comparative images) selected by the selecting unit 23. Thecompositing unit 24 performs image calculation on the composite targetimage (the composite target image selected based on the reference imageand the comparative images) selected by the selecting unit 23, accordingto the pixel shift amount detected by the detecting unit 22, so as toobtain a composite image.

More specifically, the compositing unit 24 moves the composite targetimage (the composite target image selected based on the reference imageand the comparative images) selected by the selecting unit 23 accordingto the pixel shift amount detected by the detecting unit 22, so as toobtain a composite image. The expression “moves the composite targetimage” means correcting the data of the reference image such that thecomposite target image is moved relative to the reference image. Inother words, “to move the composite target image” means extracting theimage data in which the composite target image has been moved relativeto the reference image when compositing the composite target image andthe reference image.

The compositing unit 24 moves the composite target image (the compositetarget image selected based on the reference image and the comparativeimages) relative to the reference image according to the pixel shiftamount detected by the detecting unit 22, such that the composite targetimage overlays the reference image.

The compositing unit 24 moves the composite target image (the compositetarget image selected based on the reference image and the comparativeimages), relative to the reference image in movement unit amounts(accuracy of movement) different from a detection unit amount at whichthe pixel shift amount is detected by the detecting unit 22. Forexample, the accuracy of detection of the pixel shift amount performedby the detecting unit 22 is in units of sub-pixels, whereas the accuracyof movement of the composite target image is in units of pixels. Asdescribed above, the compositing unit 24 may move the composite targetimage (the composite target image selected based on the reference imageand the comparative images) relative to the reference image at amovement pixel level (a pixel resolution for each movement, a pixelinterval for each movement, and a pixel basis for each movement)different from a detection pixel level (pixel resolution for eachdetection, a pixel interval for each detection, and a pixel basis foreach detection) at which the pixel shift amount is detected by detectingunit 22.

In accordance with the examples of (A) to (D) and (A′) to (D′) above,the pixel coordinates of the reference image and the comparative images(composite target image) after the relative movement by the compositingunit 24 are represented by the following (A″) to (D″). If the allowableerror Eallowable is ignored, the four images represented by (A″) to (D″)are equivalent to the four images (FIGS. 2A to 2C) used in themulti-shot compositing using the vibration-proof unit 80. Of course,even if the allowable error Eallowable remains, the four imagesequivalent to the images in the multi-shot compositing using thevibration-proof unit 80 are obtained.

(A″) Reference image: (dx, dy)=(0, 0)

(B″) Comparative image: (dx, dy)=(0+Eallowable, 1+Eallowable)

(C″) Comparative image: (dx, dy)=(1+Eallowable, 1+Eallowable)

(D″) Comparative image: (dx, dy)=(1+Eallowable, 0−Eallowable)

FIG. 5 is a conceptual diagram of a case in which a composite image isobtained by compositing the reference image and the comparative images 1to 3. Since the composite image obtained by the composite unit 24includes two color-information components of G (Gr, Gb) or RGBcolor-information components for each pixel, a high-definition imagewith fine detail and better color reproduction can be drawn. Further,higher-sensitivity noise can be reduced without generating moire andfalse color.

FIG. 6 is a flowchart of a process of processing an image performed bythe image processing apparatus 20. Referring to FIG. 6. the process ofprocessing an image performed by the image processing apparatus 20 isdescribed below in detail. This process of processing an image isimplemented by causing a computer, which is a component of the imageprocessing apparatus 20, to execute a predetermined program.

In step ST1, the image processing apparatus 20 receives a plurality ofimages. The plurality of images may be, for example, a plurality ofimages continuously shot by the camera unit 10, or selected andextracted from among a designated folder or cloud storage in which a setof recorded images such as moving images is stored.

In step ST2, the matching unit 21 determines whether the number of theplurality of images received by the image processing apparatus 20 hasreached a predetermined number (for example, four). When the number ofthe plurality of images has not reached the predetermined number (NO instep ST2), the process returns to step ST1 and waits until the number ofthe plurality of images received reaches the predetermined number. Whenthe number of the plurality of images has reached the predeterminednumber (YES in step ST2), the process proceeds to step ST3.

In step ST3, the matching unit 21 determines whether the plurality ofimages (the predetermined number of images) received by the imageprocessing apparatus 20 is suitable for the multi-shot compositing (forexample, the multi-shot composite mode using the camera shake). When thematching unit 21 determines that the plurality of images received by theimage processing apparatus 20 is suitable for the multi-shot compositing(YES in step ST3), the process proceeds to step ST4. When the matchingunit 21 determines that the plurality of images received by the imageprocessing apparatus 20 is not suitable for the multi-shot compositing(NO in step ST3), the process returns to step ST1 and waits for aplurality of images (a predetermined number of images) to be received bythe image processing apparatus 20 again.

In step ST4, the detecting unit 22 detects the pixel shift amount(positional shift amount) of each of the plurality of images. Thedetecting unit 22 detects the pixel shift amount, for example, using atleast one of the output of the sensor 70 and the pixel output of theimage sensor 11.

In step ST5, the selecting unit 23 selects a composite target image (acomposite target image selected using the reference image and thecomparative images) from the plurality of images according to the pixelshift amount of each of the plurality of images detected by thedetecting unit 22. The process of selecting the composite target imageperformed by the selecting unit 23 (in ST5) is further described as asub-routine in the flowcharts of FIGS. 7 and 8.

In step ST6, the compositing unit 24 obtains a composite image by movingthe composite target image (the composite target image selected usingthe reference image and the comparative images) selected by theselecting unit 23, relative to the reference image according to thepixel shift amount detected by the detecting unit 22. As a result, thecomposite target image (the composite target image selected using thereference image and the comparative images) is caused to overlay thereference image according to the pixel shift amount of each of theplurality of images detected by the detecting unit 22. Further, thepixel shift amount of (each of) the overlaid composite target image(s)is, for example, represented on a pixel-by-pixel basis as indicated bythe above (A″) to (D″). Since the composite image obtained by thecomposite unit 24 includes two color-information components of G (Gr,Gb) or RGB color-information components for each pixel, ahigh-definition image with fine detail and better color reproduction canbe drawn. Further, higher-sensitivity noise can be reduced withoutgenerating moire and false color.

Note that, when the pixel shift amount of the composite target image isthe pixel shift amount on a pixel-by-pixel basis as represented by theabove (A″) to (D″), each composite target image is preferably shifted byan odd number of pixels in at least one of the horizontal direction andthe vertical direction. For example, the pixel shift amount of 5.1 ismore preferable than the pixel shift amount of 1.5 because the pixelshift amount of 5.1 is closer to 5 pixels (odd number of pixels) thanthe pixel shift amount of 1.5. More specifically, the composite targetimage is preferably shifted by the odd number of pixels in thehorizontal direction and by the even number of pixels in the verticaldirection relative to the reference image. Alternatively, the compositetarget image is preferably shifted by the odd number of pixels in thevertical direction and by the even number of pixels in the horizontaldirection relative to the reference image. This is because, when thecomposite target image is shifted by the odd number of pixels or theeven number of pixels in both the horizontal and vertical directions,almost the same image is obtained even after the movement of thecomposite target image relative to the reference image, because theresolutions of the two color-information components of G (Gr, Gb) failto increase.

Referring to FIG. 7, a first process of selecting the composite targetimage performed by the selecting unit 23 will be described in detail.

In step ST401, the selecting unit 23 sets one of the plurality of imagesas the reference image and sets the remaining images as the comparativeimages (sets the reference image and the comparative images).

In step ST402, the selecting unit 23 initializes an allowable errorEallowable for selecting a composite target image. For example, theselecting unit 23 initializes the absolute value of Eallowable to 0.01.

In step ST403, the selecting unit 23 tries to select a composite targetimage that satisfies a predetermined condition. In this case, satisfyingthe predetermined condition means satisfying the above (A′), (B′) and/or(D′), or satisfying the above (A′) to (D′). The selecting unit 23 triesto select a composite target image that satisfies the predeterminedcondition for each combination of the reference image and eachcomparative image (the plurality of images). The selecting unit 23 triesto select the composition target image that satisfies the predeterminedcondition for plural times. When the selecting unit 23 has successfullyselected the composition target image that satisfies the predeterminedcondition (YES in step ST403), the process of selecting the compositetarget image ends. When the selecting unit 23 fails to select thecomposite target image that satisfies the predetermined condition (NO instep ST403), the process proceeds to step ST404.

In step ST404, the selecting unit 23 increases the allowable errorEallowable for selecting the composite target image. For example, theselecting unit 23 increases the absolute value of Eallowable from 0.01to 0.05.

In step ST405, the selecting unit 23 determines whether the absolutevalue of the allowable error Eallowable for selecting the compositetarget image exceeds a critical value (for example, 0.25). When theabsolute value of the allowable error Eallowable for selecting thecomposite target image exceeds the critical value (YES in step ST405),the process of selecting the composite target image ends. When theabsolute value of the allowable error Eallowable for selecting thecomposite target image falls below the critical value (NO in stepST405), the process returns to step ST403 to try select the compositetarget image that satisfies the predetermined condition using theincreased allowable error Eallowable.

Referring to FIG. 8, a second process of selecting the composite targetimage performed by the selecting unit 23 will be described in detail.

In step ST411, the selecting unit 23 sets one of the plurality of imagesas the reference image and sets the remaining images as the comparisonimage (sets the reference image and the comparative images).

In step ST412, the selecting unit 23 sets an allowable error Eallowablefor selecting a composite target image.

In step ST413, the selecting unit 23 tries to select a composite targetimage that satisfies the predetermined condition. In this case,satisfying the predetermined condition means satisfying the above (A′),(B′) and/or (D′), or satisfying the above (A′) to (D′). When theselecting unit 23 has successfully selected the composition target imagethat satisfies the predetermined condition (YES in step ST413), theprocess of selecting the composite target image ends. When the selectingunit 23 fails to select the composite target image that satisfies thepredetermined condition (NO in step ST413), the process proceeds to stepST414.

In step ST414, the selecting unit 23 re-sets the reference image and thecomparative images.

In step ST415, the selecting unit 23 determines whether all thecombinations of the reference image and each of the comparative imageshave been set. When all the combinations of the reference image and eachof the comparative images are set (YES in step ST415), the process ofselecting the composite target image ends. When all the combinations ofthe reference image and each of the comparative images are not set (NOin step ST415), the process returns to step ST413 to retry to select thecomposite target image that satisfies the predetermined condition usingthe reset reference image and comparative images.

The first selecting process of FIG. 7 and the second selecting processof FIG. 8 may be combined as appropriate. That is, when failing toselect the composite target image that satisfies the predeterminedcondition, the selecting unit 23 increases (expands) the allowable errorEallowable for selecting the composite target image and resets thereference image and the comparative images.

The digital camera according to the embodiments of the presentdisclosure has a multi-shot composite mode using the vibration-proofunit 80 (for example, micro vibration for multi-shot compositing) and amulti-shot composite mode (for example, the multi-shot composite modeusing camera shake) without using the vibration-proof unit 80. Thedigital camera includes a setting unit (for example, a setting buttonand a touch panel) for setting each shooting mode. The digital camera iscapable of issuing a warning by voice or on a display when the digitalcamera fixed to, for example, a tripod is set to the multi-shotcomposite mode in which the vibration-proof unit 80 is not used, or whenthe digital camera held by hand is set to the multi-shot composite modein which the vibration-proof unit 80 is used. Further, the digitalcamera is capable of detecting whether the digital camera is fixed to atripod. When it is determined that the digital camera is fixed to thetripod, the digital camera is automatically set to the multi-shotcomposite mode in which the vibration unit 80 is used. When it isdetermined that the digital camera is not fixed to the tripod, thedigital camera is automatically set to the multi-shot composite mode inwhich the vibration-proof unit 80 is not used.

Second Embodiment

The second embodiment of the present disclosure is described withreference to FIGS. 9 to 11. Descriptions redundant with the descriptionsof the first embodiment are omitted and only the differences aredescribed below.

FIG. 9 is a block diagram of a functional configuration of an imageprocessing apparatus (processor) 20 according to the second embodimentof the present disclosure. As illustrated in FIG. 9, the imageprocessing apparatus (processor) 20 according to the second embodimentfurther includes a dividing unit 25 in addition to the matching unit 21,the detecting unit 22, the selecting unit 23, and the compositing unit24.

The dividing unit 25 divides the plurality of images into apredetermined number of image areas (for example, corresponding imageareas). FIGS. 10A to 10D are illustrations of an example in which theplurality of images is divided into a predetermined number of imageareas. In FIG. 10A, the first image is divided into image areas 1-1,1-2, . . . 1-N in matrix each having the same size in the horizontal andvertical directions. In FIG. 10B, the second image is divided into imageareas 2-1, 2-2, . . . 2-N in matrix each having the same size in thevertical and horizontal directions. In FIG. 10C, the third image isdivided into image areas 3-1, 3-2, . . . 3-N in matrix each having thesame size in the vertical and horizontal directions. In FIG. 10D, thefourth image is divided into image areas 4-1, 4-2, . . . 4-N in matrixeach having the same size in the vertical and horizontal directions. Theblock size of each image area allows for a certain latitude. Forexample, the block size of each image area may be set to 128 pixels×128pixels.

The detecting unit 22 detects a positional shift amount (pixel shiftamount) of each of the predetermined number of image areas (for example,each of corresponding image areas) of the plurality of images, thecorresponding image areas. Referring to the example of FIGS. 10A to 10D,the detecting unit 22 detects the positional shift amount (pixel shiftamount) between the image area 1-1 of the first image, the image area2-1 of the second image, the image area 3-1 of the third image, and theimage area 4-1 of the fourth image. Further, the detecting unit 22detects the positional shift amount (pixel shift amount) between theimage area 1-2 of the first image, the image area 2-2 of the secondimage, the image area 3-2 of the third image, and the image area 4-2 ofthe fourth image. Further, the detecting unit 22 detects the positionalshift amount (pixel shift amount) between the image area 1-N of thefirst image, the image area 2-N of the second image, the image area 3-Nof the third image, and the image area of the fourth image 4-N. In thisconfiguration, the detecting unit 22 calculates the correlation betweenblocks at the same position of each image, for example, by subpixelestimation.

The selecting unit 23 selects a composite target image area from aplurality of images according to the positional shift amount (pixelshift amount), which is the correlation value detected by the detectingunit 22. For example, the selecting unit 23 sets each image area of oneof the images as the reference image area and sets the image areas ofthe other images as the comparative image areas. Then, the selectingunit 23 selects, as a composite target image area, one of thecomparative image areas based on the positional shift amount (pixelshift amount) between the reference image area and each of thecomparative image areas. Specifically, the selecting unit 23 selects acomparative image area whose positional shift amount (pixel shiftamount) is less than or equal to a predetermined threshold, whosepositional shift amount is smallest among the positional shift amountsbetween the reference image area and the comparative image areas, andwhose positional shift amount corresponds to odd number of pixels oreven number of pixels. For example, when the image areas 1-1 to 1-N ofthe first image in FIG. 10A are set as the reference image area, theselecting unit 23 selects at least one of the image areas 2-1, 3-1, 4-1as a composite target image area for the reference image area 1-1.Further, the selecting unit 23 selects at least one of the image areas2-2, 3-2, 4-2 as a composite target image area for the reference imagearea 1-2. Still further, the selecting unit 23 selects at least one ofthe image areas 2-N, 3-N, 4-N as a composite target image area for thereference image area 1-N.

The compositing unit 24 obtains a composite image based on thepositional shift amount (pixel shift amount), which is the correlationvalue detected by the detecting unit 22, and the composite target imagearea selected by the selecting unit 23. The compositing unit 24 obtainsa composite image by performing image calculation on the compositetarget image area selected by the selecting unit 23, according to thepositional shift amount (pixel shift amount) that is the correlationvalue detected by the detecting unit 22. For example, the compositingunit 24 composites or replaces the reference image area 1-1 in FIG. 10Awith the composite target image area selected from the comparative imageareas 2-1 to 4-1 in FIGS. 10B to 10D. Further, the compositing unit 24composites (replaces) the reference image area 1-2 in FIG. 10A with thecomposite target image area selected from the comparative image areas2-2 to 4-2 in FIGS. 10B to 10D. Further, the compositing unit 24composites (replaces) the reference image area 1-N in FIG. 10A with thecomposite target image area selected from the comparative image areas2-N to 4-N in FIGS. 10B to 10D.

As a result, the compositing unit 24 performs image calculation(composition or replacement) on the composite target image areasobtained by the detecting unit 22 and the selecting unit 23 incooperation with each other for each of the plurality of image areasdivided by the dividing unit 25.

That is, each reference image area of one reference image is compositedor replaced with a composite target image area selected from comparativeimage areas of the comparative images. For example, the reference imagearea 1-1 of the first image (the reference image) is composited orreplaced with the composite target image area 2-1 of the second image,and the reference image area 1-2 of the first image is composited orreplaced with the composite target image area 3-2 of the third image.Further, the reference image area 1-N of the first image is compositedor replaced with a composite target image area 4-N of the fourth image.

When the selecting unit 23 fails to select a composite target image areafrom the comparative image areas of the comparative images for a certainreference image of the reference image, the reference image area as isis used without the composition or replacement of the reference imagearea.

FIG. 11 is a flowchart of an image-capturing process according to asecond embodiment of the present disclosure.

In step ST11, the dividing unit 25 divides a plurality of images into apredetermined number of image areas.

In step ST12, the detecting unit 22 detects the positional shift amount(pixel shift amount) of each of the predetermined number of image areasof the plurality of images.

In step S13, the selecting unit 23 selects a composite target image areafrom the plurality of images according to the positional shift amount(pixel shift amount) that is the correlation value detected by thedetecting unit 22.

In step ST14, it is determined whether a composite target image area hasbeen selected from all sets of image areas. When the composite targetimage area has not been selected from all sets of image areas (NO instep ST14), the process returns to step ST13 to repeat the loop of stepST13 and step ST14 until the composite target image area is selected forall the image areas. When the combination target image area is selectedfrom each set of the image areas (YES in step ST14), the processproceeds to step ST15.

In step ST15, the compositing unit 24 obtains a composite image based onthe positional shift amount (pixel shift amount), which is thecorrelation value detected by the detecting unit 22, and the compositetarget image area selected by the selecting unit 23.

In the second embodiment described above, a plurality of images isdivided into a predetermined number of image areas, and the positionalshift amount of each of the predetermined number of image areas in theplurality of images is detected. Then, a composite target image area isselected from the plurality of images based on the positional shiftamount, and a composite image is obtained based on the positional shiftamount and the composite target image area With the configurationaccording to the second embodiment of the present disclosure,higher-quality image with high detail and less moiré, false color, andhigh sensitivity noise can be provided as compared to the configurationaccording to the first embodiment in which the positional shift amountis detected on an image-by-image basis and a composite target image isselected to obtain a composite image.

Third Embodiment

The digital camera according to the first and second embodiments doesnot drive (for example, image blur (vibration) correction drive) amoving member (for example, the image sensor 11) using thevibration-proof unit 80 in the multi shot composite mode. However, whenthe image blur correction drive is performed crudely while using thevibration-proof unit 80 without perfectly correcting the positionalshift of a plurality of images (images are not perfectly aligned at aspecific position), the image blur correction drive is executed usingthe vibration-proof unit 80.

That is, executing the image blur correction drive using vibration-proofunit 80 still fails to completely eliminate image blur (the image ismisaligned (shifted) on the order of several microns). Accordingly, inthe configuration according to the third embodiment, such an image shift(misalignment) is used in the multi-shot compositing. This configurationis based on the concept that the amount of drive in the image blurcorrection drive using the vibration-proof unit 80 is significantlylarger than the positional shift amount (pixel shift amount) of eachimage used in the multi-shot compositing.

In the third embodiment, a plurality of images is obtained by, forexample, continuous shooting after setting the multi-shot composite mode(the multi-shot composite mode using camera shake, with image blurcorrection drive using the image stabilizing unit 80). Then, onecomposite image is obtained by image composite processing based on theplurality of images.

For example, as in the first embodiment, the configuration according tothe third embodiment can detect the pixel shift amounts of a pluralityof images, set any one of the plurality of images as a reference image,and set the remaining images as comparative images. Further, theconfiguration can select a composite target image from the comparativeimages based on the pixel shift amount between the reference image andeach of the comparative images, and move the composite target imagerelative to the reference image based on the positional shift amount(pixel shift amount) to obtain a composite image.

Alternatively, as in the second embodiment, the configuration accordingto the third embodiment can divide a plurality of images into apredetermined number of image areas, and detect positional shift amountof each of the predetermined number of image areas in the plurality ofimages. Further, the configuration according to the third embodiment canselect a composite target image area from the plurality of images basedon the positional shift amounts.

The configuration of the vibration-proof unit 80 is described in detailwith reference to FIGS. 12A, 12B, 13, 14, and 15. In each figure, afirst direction (Z direction and Z-axis direction) is parallel to theoptical axis O of the imaging optical system and a second direction (Xdirection and X-axis direction) is orthogonal to the first direction.Further, a third direction (Y direction and Y-axis direction) isorthogonal to both the first direction and the second direction. Forexample, assuming that the X axis, the Y axis, and the Z axis arecoordinate axes in a three-dimensional orthogonal coordinate system,when the optical axis O is designated as the Z axis, the X axis and theY axis are orthogonal to each other and both are orthogonal to theX-axis. When the digital camera is disposed in the normal position(horizontal position), the first direction (the Z direction, the Z axis,the optical axis O) and the second direction (the X direction and the Xaxis) are along the horizontal direction of the digital camera, and thethird direction (the Y direction and the Y-axis) are along the verticaldistance of the digital camera

The digital camera includes, as a unit for detecting vibration(fluctuation) of a camera body CB, a roll (tilt (rotation) around theZ-axis) detecting unit, a pitch (tilt (rotation) around the X-axis)detecting unit, a yaw (tilt (rotation) around the Y-axis) detectingunit, an X-direction acceleration detecting unit, a Y-directionacceleration detecting unit, and a Z-direction acceleration detectingunit. Each detection unit includes a 6-axis sensor or a set consistingof a 3-axis gyro sensor and a 3-axis acceleration sensor. In someembodiments, each detecting unit may constitute the sensor 70 in FIG. 1.

An imaging block (for example, the camera unit 10 in FIG. 1) includes animage sensor 110 and a stage device 120 that supports the image sensor110. The stage device 120 includes a movable stage 121 on which theimage sensor 110 is mounted, a front stationary yoke 122 on the front ofthe movable stage 121, and a rear stationary yoke 123 on the back of themovable stage 121. The stage device 120 is capable of moving up themovable stage 121 (moved up against gravity and kept at rest) relativeto the front and rear stationary yokes 122 and 123 at least when madeconductive. The stage device 120 is capable of moving the movable stage121 in a floating state (moved up) along the Z direction (firstdirection) (parallel movement in the Z direction), along the X direction(second direction) (parallel movement in the X direction) orthogonal tothe Z direction, and along the Y direction (third direction) (parallelmovement in the Y direction) orthogonal to both the Z direction and theX direction. Further, the stage device 120 is capable of causing themovable stage 121 in a floating state (moved up) to tilt (rotate) aroundthe X-axis (second direction), around the Y-axis (third direction), andaround the Z-axis (first direction). That is, the movable stage 121 ismovable with six degrees of freedom, along 6 axes.

The body CPU (for example, the CPU 90 in FIG. 1) calculates thedirection of blur and the blur speed of the digital camera based onpitch (tilting (rotation) in the X direction), yaw (tilting (rotation)in the Y direction), roll (tilting (rotation) in the Z direction), theX-direction acceleration, the Y-direction acceleration, and theZ-direction acceleration. The body CPU calculates, for example, thedrive direction, the drive speed, the drive amount of drive of the imagesensor 110 to prevent an image projected onto the image sensor 110 frommoving relative to the image sensor 110. Based the calculation results,the CPU causes the stage device 120 to travel in parallel, tilt, travelin parallel while tilting, travel in parallel after tilting, and tiltafter traveling in parallel.

The stage device 120 holds the movable stage 121, to which the imagesensor 110 is fixed, such that the movable stage 121 freely travels inparallel, tilts, travels in parallel while tilting, and travels inparallel after tilting relative to the front stationary yoke 122 and therear stationary yoke 123. The movable stage 121 is a rectangular platemember larger than the image sensor 110 when viewed from the front. Thefront stationary yoke 122 and the rear stationary yoke 123 arerectangular frame members each having the same shape and an outer shapelarger than the movable stage 121 in plan view. Each of the frontstationary yoke 122 and the rear stationary yoke 123 has a rectangularopening (122 a/123 a) larger than the outer shape of the image sensor110 at the central portion of each of the front stationary yoke 122 andthe rear stationary yoke 123, when viewed from the front (the Z.direction).

The front stationary yoke 122 has an X-direction magnet MX on at leastone side of the right and left (X direction) of the opening 122 a withrespect to the Z-axis with the Y-axis as the center line on the back(the surface opposite to the object side). However, in the embodiment asillustrated in FIGS. 12A and 12B, an X-direction magnet MX is disposedon each side of the right and left of the opening 122 a. That is, a pairof X-direction magnets MX, each made of a permanent magnet having thesame specification, is fixed to the back surface of the front stationaryyoke 122. By passing the magnetic flux of the X-direction magnets MXthrough the front stationary yoke 122 and the rear stationary yoke 123,a magnet circuits that generates thrust in the X direction (the seconddirection) is formed between the X-direction magnets MX on the right andleft sides and the opposed portion of the rear stationary yoke 123.

The front stationary yoke 122 has a pair of a Y-direction magnet MYA anda Y-direction magnet MYB at the lower side relative to the opening 122 aon the back of the front stationary yoke 122. The magnet MYA and themagnet MYB are opposed to each other across the Y-axis as the centerline and away from the Z-axis. Each of the magnet MYA and the magnet MYBis permanent magnet having the same specification. By passing themagnetic flux of the magnet MYA and the magnet MYB through the frontstationary yoke 122 and the rear stationary yoke 123, a magnet circuitthat generates thrust in the Y direction (the third direction) is formedbetween the magnet yoke 122 and the magnet yoke 123.

The front stationary yoke 122 also has Z-direction magnets MZA, MZB andMZC fixed onto three positions away from the Y-direction magnets MYA andMYB on the back surface. The magnets MZA, MZB and MZC are permanentmagnets of the same specification. The three Z-direction magnets MZA,MZB, and MZC are disposed at substantially equal intervals in a planeorthogonal to the Z-axis with the Z axis as the center of the plane. Bypassing through the Z-direction magnets MZA, MZB and MZC through thefront stationary yoke 122 and the rear stationary yoke 123, a pluralityof magnet circuits that generates thrust in the Z direction (the firstdirection) is formed between the Z-direction magnets MZA, MZB and MZCand the rear stationary yoke 123.

The movable stage 121 has a hole 121 a for the image sensor 110 at thecenter portion of the movable stage 121. The hole 121 a is rectangularwhen viewed from the front. The image sensor 110 is fit in the hole 121a. The image sensor 110 projects forward beyond the hole 121 a in thedirection of the optical axis O of the movable stage 121.

The movable stage 121 further has a pair of X-drive coils CX and a pairof a Y-drive coil CYA and a Y-drive coil CYB. The X-drive coils CX arefixed onto the outer portions of the right and left sides (short sides)of the image sensor 110, respectively. The Y-drive coil CYA and theY-drive coil CYB are fixed onto the lower portion of image sensor 110(in the vicinity of the lower side (long side) of the image sensor 110),apart from each other along the right-to-left direction of the imagesensor 110. The movable stage 121 further has a circular Z-drive coilCZA and a pair of circular Z-drive coils CZB and CZC. The Z-drive coilCZA is stationary (in the intermediate position) between the Y-drivecoils CYA and CYB. The Z-drive coils CZB and CZC are stationary at theupper position relative to the pair of the X-drive coils CX.

The above-described X-drive coil CX, the Y-drive coil CYA, the Y-drivecoil CYB, the Z-drive coil CZA, the Z-drive coil CZB, and the Z-drivecoil CZC are connected to an actuator drive circuit that controls powerdistribution.

In the movable stage 121, X-direction Hall elements HX are fixed in theair core areas of the X-drive coils CX, and a Y-direction Hall elementHYA and a Y-direction Hall element HYB are fixed in the air core areasof the Y-drive coils CYA and CYB, respectively. Further, Z-directionHall elements HZA, HZB, and HZC are fixed in the air core areas ofZ-drive coils CZA, CZB, and CZC, respectively.

A position detection circuit detects the position of the movable stage121 in the X direction, the position in the Y direction, the position inthe Z direction, the position of tilt rotation around the X-axis (tiltrotation angle around the X-axis and pitch angle), the position of tiltrotation around the Y-axis (tilt rotation angle around the Y-axis andYaw angle), and the position of tilt rotation around the Z-axis (tiltrotation angle around the Z-axis and roll angle), based on detectionsignals output from X-direction Hall elements HX, the Y-direction Hallelements HYA and HYB, and Z-direction Hall elements HZA, HZB, and HZC.

Based on the detection result of the position detection circuit, theactuator drive circuit drives the image sensor 110 (the movable stage121) by controlling power distribution to the X-drive coils CX, theY-drive coils CYA, CYB, CZA and the Z-drive drive coils CZA, CZB, andCZC. For example, the vibration-proof unit 80 serves as a camera shakecorrection device (drive device) that corrects image blur (vibration) bydriving (moving) the image sensor 110, which is a part of theimage-capturing device, as a drive member in a direction different fromthe direction of the optical axis O (Z-axis) of the image-capturingdevice. Note that the drive member to be driven is not limited to theimage sensor 110, and may be, for example, an image-blur correction lensas a part of the photographing lens.

The present inventor has conceived of the following concept throughintensive studies of a technique of executing multi-shot composite whileexecuting image-blur correction drive using the above-described hexaxialdrive unit (however, the mode of image-blur correction is not limited)as one example. Even if a parallel-direction shift of the drive member(image sensor) remains within a plane (XY plane) orthogonal to theoptical axis O (Z axis), the image quality of the multi-shot compositeis not adversely affected. However, it is found that if arotational-direction shift of the drive member (image sensor) remainswithin the plane (XY plane) orthogonal to the optical axis O (Z axis),the image quality of the multi-shot composite is adversely affected.

As described above, in the embodiments of the present disclosure, theimage calculation such as detection of the positional shift amount(pixel shift amount) of a plurality of images or image areas isperformed based on the XY coordinate axes in the XY plane. Accordingly,when a rotational shift within the XY plane is large, correlationbetween a plurality of images or between a plurality of image areascannot be obtained, and appropriate image calculation may be difficult.

FIGS. 16A and 16B are illustrations of adverse effects of image blur(shift, vibration) in the rotational direction within the XY plane. Asillustrated in FIGS. 16A and 16B, the image blur amount in therotational direction within the XY plane decreases with a reduction indistance to the optical axis O (Z-axis) (closer to the center of theimage), and increases with an increase in distance to the optical axis O(Z-axis) (closer to the periphery of the image).

In the embodiments of the present disclosure, not only the shift amountin the parallel direction within a plane (the XY plane) orthogonal tothe optical axis O (Z axis) but also the shift amount in the rotationaldirection within the plane orthogonal to the optical axis O (Z axis)(the XY plane) is corrected using the vibration-proof unit 80. With sucha configuration, the accuracy of the image calculation can be increasedand the image quality of the multi-shot composite can be improved aswell. Further, the processing load and the processing time of the imagecalculation can be reduce.

In some embodiments, the vibration-proof unit (drive device) 80 mayrelatively reduce the drive component (drive amount) of the drive member(image sensor) the parallel direction within a plane (XY plane)orthogonal to the optical axis O (Z axis), and relatively increase thedrive component (drive amount) of the drive member (image sensor) in therotational direction within the plane (XY plane) orthogonal to theoptical axis O (Z axis). This configuration permits a certain amount ofparallel-direction shift components (shift amount) of the driving member(image sensor) to remain in the XY image, which has a small adverseeffect on the image quality of the multi-shot composite. Further, such aconfiguration positively eliminates the rotational-direction shiftcomponents (shift amount) of the driving member (image sensor) toprevent a significant adverse effect on the image quality to increasethe image quality of the multi-shot composite.

Further, as in the second embodiment, by dividing a plurality of imagesinto a predetermined number of image areas by the dividing unit 25 andcalculating a positional shift amount (pixel shift amount) for eachimage area, the shift amount of drive member (image sensor) in therotational direction can be reduced.

In this case, the image areas divided by the dividing unit 25 preferablyhave different sizes. More specifically, among the image areas dividedby the dividing unit 25, the center portion of each of the plurality ofimages preferably has a relatively large size, and each image area ofthe peripheral portion of each of the plurality of images preferably hasa relatively small size.

FIG. 17 is an illustration of an example in which a plurality of imagesis divided into image areas having different sizes. In FIG. 17, theimage area is constituted by a total of 80 blocks in the minimum blockunit, that is, eight blocks in the vertical direction×ten blocks in thehorizontal direction. The image area in FIG. 17 is divided into amaximum image area block in the center portion of the image area, twointermediate image area blocks on each side of the maximum image area,and minimum image area blocks on the periphery of the image area,surrounding the maximum image area block and the intermediate image areablocks. The maximum image area block has a size of 16 (four-by-fourpixels) minimum image area blocks (minimum block unit). The intermediateimage area block has a size of 4 (two-by-two pixels) minimum image areablocks (minimum block unit).

For example, when there is a shift in the rotation direction among aplurality of images, the shift amount decreases toward the centerportion of the image, and increases toward the periphery of the image(see FIGS. 16A and 16B). In view of this, the image area correspondingto the center portion of the image in which the shift amount in therotational direction is small is divided into large (coarse) blocks,while the image area in the periphery of the image in which the shiftamount in the rotational direction is large is divided into small (fine)blocks. Accordingly, the accuracy of image calculation in each imagearea block (particularly in the image area blocks in the periphery ofthe image) can be increased, and image quality of the multi-shotcomposite can be improved. Further, the processing load and theprocessing time of the image calculation can be reduced. In FIG. 17, ifall the image area blocks are divided into the minimum image area blocks(minimum block units), the processing load of the image calculation andthe processing time increase. Further, in FIG. 17, if all the image areablocks are divided into the maximum image area blocks, correlationbetween the image area blocks might not be obtained (the pixel shiftamount might not be calculated) in the image peripheral portion in whichthe shift amount in the rotation direction is large. Any one of theabove-described operations may be performed in various other ways, forexample, in an order different from the one described above.

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Processing circuitryincludes a programmed processor, as a processor includes circuitry. Aprocessing circuit also includes devices such as an application specificintegrated circuit (ASIC), DSP (digital signal processor), FPGA (fieldprogrammable gate array) and conventional circuit components arranged toperform the recited functions.

Although the embodiments of the present disclosure have been describedabove, the present disclosure is not limited to the embodimentsdescribed above, but a variety of modifications can naturally be madewithin the scope of the present disclosure. Numerous additionalmodifications and variations are possible in light of the aboveteachings. It is therefore to be understood that within the scope of theappended claims, the embodiments may be practiced otherwise than asspecifically described herein. For example, elements and/or features ofdifferent illustrative embodiments may be combined with each otherand/or substituted for each other within the scope of this disclosureand appended claims.

1-17. (canceled)
 18. An apparatus, comprising: processing circuitryconfigured to detect a positional shift of each of a plurality of imagescaptured by an image-capturing device, based on shake of an image sensordetected by a sensor; and obtain a composite image based on the detectedpositional shift.
 19. The apparatus according to claim 18, wherein theprocessing circuitry is further configured to: set one of the pluralityof images as a reference image and set remaining images of the pluralityof images as comparative images; compare the reference image and each ofthe comparative images; detect a pixel shift amount between thereference image and each of the comparative images based on results ofthe comparing; select a composite target image from the plurality ofimages based on the detected positional shift and a composite targetimage from the comparative images based on the detected pixel shiftamount; and move the composite target image relative to the referenceimage based on the positional shift to obtain the composite image. 20.The apparatus according to claim 19, wherein the processing circuitry isfurther configured to detect the pixel shift amount of each of theplurality of images on one of a pixel-to-pixel basis and asubpixel-to-subpixel basis, based on a pixel output of the image sensor.21. The apparatus according to claim 20, wherein the processingcircuitry is further configured to exclude a pixel output to be used inan operation other than an image-capturing operation, from the pixeloutput of the image sensor.
 22. The apparatus according to claim 19,wherein the processing circuitry is further configured to detect thepixel shift amount of each of the plurality of images for each RGBplane, based on a pixel output of the image sensor.
 23. The apparatusaccording to claim 22, wherein the processing circuitry is furtherconfigured to detect the pixel shift amount of each of the plurality ofimages with a change in RGB plane to be used.
 24. The apparatusaccording to claim 19, wherein the processing circuitry is furtherconfigured to move the composite target image relative to the referenceimage to have the composite target image overlay the reference image.25. The apparatus according to claim 19, wherein the processingcircuitry is further configured to move the composite target imagerelative to the reference image in units of movement different fromunits of detection at which the pixel shift amount is detected.
 26. Theapparatus according to claim 18, wherein the processing circuitry isfurther configured to: divide the plurality of images into apredetermined number of image areas; detect a positional shift for eachof the predetermined number of image areas; select a composite targetimage area from the plurality of images, based on the detectedpositional shift; and obtain the composite image based on the positionalshift and the selected composite target image area
 27. The apparatusaccording to claim 26, wherein the processing circuitry is furtherconfigured to divide the plurality of images into an image area having alargest size at a central portion of each of the plurality of images andperipheral image areas each having a smallest size at a peripheralportion of each of the plurality of images.
 28. The apparatus accordingto claim 18, wherein the plurality of images includes continuouslycaptured images, and the obtained composite image includes twocolor-information components of G (Gr, Gb) or RGB color-informationcomponents for each pixel.
 29. The apparatus according to claim 18,further comprising: the image-capturing device configured to capture theplurality of images, the image-capturing device including the imagesensor; and the sensor configured to detect shake of the image sensor.30. The apparatus according to claim 29, wherein the apparatus is adigital camera.
 31. The apparatus according to claim 29, wherein theapparatus is a mobile phone.
 32. A method of processing an image, themethod comprising: detecting a positional shift of each of a pluralityof images captured by an image-capturing device, based on shake of animage sensor detected by a sensor; and obtaining a composite image basedon the detected positional shift.
 33. The method according to claim 32,wherein the plurality of images obtained in the obtaining step includescontinuously captured images, and the obtained composite image includestwo color-information components of G (Gr, Gb) or RGB color-informationcomponents for each pixel.
 34. A non-transitory recording medium storinga program for causing a computer to execute a method of processing animage, the method comprising: detecting a positional shift of each of aplurality of images captured by an image-capturing device, based onshake of an image sensor detected by a sensor; and obtaining a compositeimage based on the detected positional shift.
 35. The non-transitoryrecording medium according to claim 34, wherein the plurality of imagesincludes continuously captured images, and the obtained composite imageincludes two color-information components of G (Gr, Gb) or RGBcolor-information components for each pixel.
 36. The apparatus of claim18, further comprising a memory that stores a program for causing theprocessing circuitry to detect the positional shift and to obtain thecomposite image.